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Background 

The invention relates to expanding an effective 
vocabulary of a speech recognition system. 

A speech recognition system analyzes a user's speech 
to determine what the user said. Most speech recognition 
systems are frame-based. In a frame-based system, a 
processor divides a signal descriptive of the speech to be 
recognized into a series of digital frames, each of which 
corresponds to a small time increment of the speech. 

A speech recognition system may be a "discrete" 
system that recognizes discrete words or phrases but which 
requires the user to pause briefly between each discrete 
word or phrase. Alternatively, a speech recognition system 
may be a "continuous" system that can recognize spoken words 
or phrases regardless of whether the user pauses between 
them. Continuous speech recognition systems typically have 
a higher incidence of recognition errors in comparison to 
discrete recognition systems due to complexities of 
recognizing continuous speech. A detailed description of 
continuous speech recognition is provided in U.S. Patent No. 
5,202,952, entitled "LARGE -VOCABULARY CONTINUOUS SPEECH 
PREFILTERING AND PROCESSING SYSTEM, " which is incorporated 
by reference. 

In general, the processor of a continuous speech 
recognition system analyzes "utterances" of speech. An 
utterance includes a variable number of frames and 
corresponds, for example, to a period of speech followed by 
a pause of at least a predetermined duration. 

The processor determines what the user said by 
finding acoustic models that best match the digital frames 
of an utterance, and by identifying text that corresponds to 



those acoustic models. An acoustic model may correspond to 
a word, phrase or command from a vocabulary. An acoustic 
model also may represent a sound, or phoneme, that 
corresponds to a portion of a word. Collectively, the 
5 constituent phonemes for a word represent the phonetic 

spelling of the word. Acoustic models also may represent 
silence and various types of environmental noise. In 
general, the processor may identify text that corresponds to 
the best-matching acoustic models by reference to phonetic 
10 word models in an active vocabulary of words and phrases. 

The words or phrases corresponding to the best 
matching acoustic models are referred to as recognition 
candidates. The processor may produce a single recognition 
candidate for an utterance, or may produce a list of 
[y 15 recognition candidates. 

Summary 

The invention provides techniques for creating and 
yi using fragmented word models to increase the effective size 

T' of an active vocabulary of a speech recognition system. The 

O 2 0 active vocabulary represents all words and word fragments 

that the speech recognition system is able to recognize. 
•-•^^ Each word may be represented by a combination of acoustic 



models. As such, the active vocabulary represents the 
combinations of acoustic models that the speech recognition 

25 system may compare to a user's speech to identify acoustic 
models that best match the user's speech. 

Memory and processing speed requirements tend to 
increase with the number of entries in the active 
vocabulary. As such, the size of the active vocabulary that 

30 may be processed in an allotted time by a particular 

processor using an available amount of memory is limited. 
Since the recognizer does not recognize words that are not 
included in the active vocabulary, the ability of the 
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recognizer to recognize less-frequently-used words may be 
improved by increasing the size of the active vocabulary. 

The effective size of the active vocabulary may be 
increased by dividing words into constituent components or 
5 fragments (for example, prefixes, suffixes, separators, 
infixes, and roots) and including each component as a 
separate entry in the active vocabulary. Thus, for example, 
a list of words and their plural forms (for example, "book, 
books, cook, cooks, hook, hooks, look and looks") may be 
10 represented in the active vocabulary using the words (for 
example, "book, cook, hook and look") and an entry 
representing the suffix that makes the words plural (for 
example, "+s", where the "+" preceding the "s" indicates 
that "+s" is a suffix) . For a large list of words, and 
iy 15 ignoring the entry associated with the suffix, this 

technique may reduce the number of vocabulary entries needed 
to represent the list of words considerably. 
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H The invention provides a method of expanding an 

effective active vocabulary of a speech recognition system 
Q 2 0 that uses a speech recognizer. The speech recognizer 
:2 perform speech recognition on a user utterance to produce 

^ one or more recognition candidates. Speech recognition 

includes comparing digital values representative of the user 
utterance to a set of acoustic models representative of an 
25 active vocabulary of the system. The set of acoustic models 
includes models of words and models of word fragments. 

The method further includes receiving the 
recognition candidates from the speech recognizer. When a 
received recognition candidate includes a word fragment, the 
3 0 method includes determining whether the word fragment may be 
combined with one or more adjacent word fragments or words 
to form a proposed word included in a backup dictionary of 
the speech recognition system. 
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Furthermore, if the word fragment may be combined 
with one or more adjacent word fragments or words to form a 
proposed word included in a backup dictionary of the speech 
recognition system, the method includes modifying the 
5 recognition candidate to substitute the proposed word for 
the word fragment and the one or more adjacent word 
fragments or words used to form the proposed word. 

Moreover, if the word fragment may not be combined 
with one or more adjacent word fragments or words to form a 
10 proposed word included in a backup dictionary of the speech 
recognition system, the method includes discarding the 
recognition candidate. 

Embodiments may include one or more of the following 
features. For example, the expanded effective vocabulary 
Q 15 may include words from the backup dictionary that are formed 
from a combination of words and word fragments or word 
fragments and word fragments from an active vocabulary that 
includes words and word fragments, and words from the active 
vocabulary. 

p 2 0 The word fragments may include suffixes, prefixes, 

25 and roots that are not words. Additionally, one or more 

spelling rules may be associated with each prefix and each 
v3 suffix. Determining whether the word fragment may be 

combined with one or more adjacent word fragments or words 
2 5 to form a proposed word may include using a prefix or suffix 
as the particular word fragment and using an associated 
spelling rule in forming the proposed word. As a result of 
using the associated spelling rule, a spelling of the 
proposed word may differ from a spelling that would result 
30 from merely concatenating the particular word fragment with 
the one or more adjacent word fragments or words. 

Determining whether the word fragment may be 
combined with one or more adjacent word fragments or words 
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may include retrieving from the received recognition 
candidate a sequence that includes the particular word 
fragment and adjacent word fragments or words. Determining 
may further include determining if the sequence is a valid 
5 sequence . 

A valid sequence may include only one or more 
allowed adjacent combinations of word fragments and words. 
Moreover, allowed adjacent combinations may include one or 
more prefixes, followed by a root or a word, followed by one 
10 or more suffixes. Other allowed adjacent combinations may 
include a root or a word followed by one or more suffixes, 
and one or more prefixes followed by a root or a word. 
The method may further include combining the 
^-^3 particular word fragment with the one or more adjacent word 

t^ti 15 fragments or words to form a second proposed word that 
^^5 differs from the first proposed word by using a second 

Q 

l^, associated spelling rule in forming the proposed word. 

4 One or more spelling rules may be associated with a 
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particular word fragment. And, combining the particular 
2 0 word fragment with one or more adjacent word fragments or 



words to form a proposed word may include using an 



associated spelling rule in forming the proposed word. As a 
result of using the associated spelling rule, a spelling of 
the proposed word may differ from a spelling that would 
25 result from merely concatenating the particular word 

fragment with the one or more adjacent word fragments or 
words . 

Determining whether the word fragment may be 
combined with one or more adjacent word fragments or words 
3 0 to form a proposed word included in a backup dictionary of 
the speech recognition system may include searching the 
backup dictionary for the proposed word. 
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Modifying the recognition candidate may include 
forming a prospective recognition candidate, and if the 
prospective recognition candidate includes an additional 
word fragment, forming a final recognition candidate. The 
5 prospective recognition candidate may be formed by modifying 
the recognition candidate to substitute the proposed word 
for the word fragment and the one or more adjacent word 
fragments or words used to form the proposed word. 
Moreover, the prospective recognition candidate may be 
10 further processes to generate an additional word using the 
additional word fragment and one or more adjacent words or 
word fragments. The final recognition candidate may be 
formed by replacing the additional word fragment and the one 
or more adjacent words with the additional word. 

%Q 

ly 15 A score may be associated with the received 

recognition candidate, such that the method further includes 
ly producing a score associated with the modified recognition 

candidate by rescoring the modified recognition candidate. 

The score associated with the received recognition 
2 0 candidate may include an acoustic component and a language 
model component. Rescoring the modified recognition 
%^ candidate may therefore include generating a language model 

score for the modified recognition candidate. 

Producing the score associated with the modified 
25 recognition candidate may include combining the acoustic 
component of the score for the received recognition 
candidate with the language model score generated for the 
modified recognition candidate. 

Rescoring the modified recognition candidate may 
30 include generating an acoustic model score for the modified 
recognition candidate. Furthermore, producing the score 
associated with the modified recognition candidate may 
include combining the acoustic model score generated for the 
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modified recognition candidate with the language model score 
generated for the modified recognition candidate. 

The score associated with the received recognition 
candidate may include an acoustic component and a language 
model component. Rescoring the modified recognition 
candidate may therefore include generating an acoustic score 
for the modified recognition candidate. 

Producing the score associated with the modified 
recognition candidate may include combining the language 
model component of the score for the received recognition 
candidate with the acoustic score generated for the modified 
recognition candidate. 

The method may include generating the acoustic model 
of a word fragment by initially comparing a word of the 
active vocabulary to a similar word of a backup dictionary 
to identify a word fragment that may be used to convert the 
word of the active vocabulary to the word of the backup 
dictionary. Then, the acoustic model of the word fragment 
may be generated using a portion of an acoustic model of the 
word of the backup dictionary that is not included in an 
acoustic model of the word of the active vocabulary. 

Comparing a word of the active vocabulary to a 
similar word of a backup dictionary may include comparing 
spellings of the two words. 

The method may include generating the acoustic 
models of word fragments by initially comparing words of the 
active vocabulary to similar words of a backup dictionary to 
identify spelling rules that may be used to convert the 
words of the active vocabulary to words of the backup 
dictionary. Then, the spelling rules may be employed in 
identifying word fragments. 

Employing the spelling rules in identifying the word 
fragments may include grouping spelling rules together to 



form possible affixes, the affixes including prefixes and 
suffixes. Then, words of the backup dictionary may be 
analyzed using the affixes to identify roots that may be 
combined with the affixes to produce words of the backup 
dictionary . 

Acoustic models for the roots may be generated using 
portions of acoustic models of the words of the backup 
dictionary that are not included in acoustic models of the 
affixes . 

Affixes and roots may be added to the active 
vocabulary as word fragments. A set of spelling rules 
associated with an affix may be stored in the active 
vocabulary. 

A language model associated with the active 
vocabulary may be created. 

Creating the language model may include retrieving a 
training collection of text, the training collection of text 
including words from the backup dictionary and words from 
the active vocabulary. Creating the language model may then 
include modifying the training collection of text by 
replacing any splittable backup dictionary words with their 
corresponding words and word fragments. Then, language 
model scores for words and word fragments of the active 
vocabulary may be generated using the modified collection of 
text. 

Creating the language model may include creating an 
N-gram language model that includes retrieving a training 
collection of text. The training collection of text 
includes words from the backup dictionary and words from the 
active vocabulary. Creating the N-gram language model may 
then include determining a frequency of each N-gram word 
sequence that appears in the training collection of text. 
Then, the N-gram word sequences may be modified by replacing 



any splittable backup dictionary words with their 
corresponding words and word fragments. Based on the N-gram 
word sequence frequencies, a frequency of each modified N- 
gram sequence that includes words, word fragments, or words 
5 and word fragments may be determined. Creating the N-gram 
language model may further include generating an N-gram 
language model for the words and word fragments of the 
active vocabulary based on the N-gram word and word fragment 
sequence frequencies . 
10 Word fragments may include syllables. Each syllable 

may include a unit of spoken language. The unit of spoken 
language includes a single uninterrupted sound formed by a 
vowel or diphthong alone, of a syllabic consonant alone, or 
of either with one or more consonants. A syllable may 
Ly' 15 include a vowel and one or more consonants clustered around 
the vowel . 

y Determining whether the word fragment may be 
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combined with one or more adjacent word fragments or words 
to form a proposed word included in the backup dictionary 
^3 2 0 may include searching the backup dictionary for the proposed 

word based on a pronunciation of the proposed word. 
M In an alternate embodiment, the invention provides a 

method of recognizing speech. The method includes using a 
speech recognizer to perform speech recognition on a user 
25 utterance to produce one or more recognition candidates. 
The speech recognition includes comparing digital values 
representative of the user utterance to a set of acoustic 
models representative of an active vocabulary of the system, 
the set of acoustic models including models of words, models 
3 0 of roots that are not words, and models of affixes that are 
not words, the affixes including prefixes and suffixes. The 
method further includes receiving the recognition candidates 
from the speech recognizer. When a received recognition 
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candidate includes an affix, the method includes combining 
the affix with one or more adjacent words, roots, or other 
affixes to form a new word. Then, the recognition candidate 
is modified to substitute the new word for the affix and the 
one or more adjacent words, roots, or other affixes used to 
form the new word. 

In another embodiment, the invention provides a 
method of generating an acoustic model of a word fragment. 
The method includes comparing a word of an active vocabulary 
to a similar word of a backup dictionary to identify a word 
fragment that may be used to convert the word of the active 
vocabulary to the word of the backup dictionary. The method 
further includes generating the acoustic model of the word 
fragment using a portion of an acoustic model of the word of 
the backup dictionary that is not included in an acoustic 
model of the word of the active vocabulary. 

Embodiments may include one or more of the following 
features. For example, comparing a word of the active 
vocabulary to a similar word of a backup dictionary may 
include comparing spellings of the two words. 

The invention provides a method of generating 
acoustic models of word fragments. The method includes 
comparing words of an active vocabulary to similar words of 
a backup dictionary to identify spelling rules that may be 
used to convert the words of the active vocabulary to words 
of the backup dictionary. Then, the method employs the 
spelling rules in identifying word fragments. 

Embodiments may include one or more of the following 
features. For example, employing the spelling rules in 
identifying word fragments may include forming possible 
affixes, and analyzing words of the backup dictionary using 
the affixes to identify roots that may be combined with the 
affixes to produce words of the backup dictionary. Possible 
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and word fragments may be determined. Creating the N-gram 
language model may further include generating an N-gram 
language model for the words and word fragments of the 
active vocabulary based on the N-gram word and word fragment 
5 sequence frequencies . 

Other features and advantages will become apparent 
from the following description, including the drawings, and 
from the claims. 
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Description of the Drawings 
10 Fig. 1 is a block diagram of a speech recognition 

system. 

Fig. 2 is a block diagram of speech recognition 
software of the system of Fig. 1. 

Fig. 3 is a flow chart of a signal processing 
15 procedure performed by the software of Fig. 2. 

Figs. 4A and 4B are state diagrams of a constraint 
grammar . 

Fig. 5 is a graph of a lexical tree. 
Fig. 6 is a graph of a portion of the lexical tree 



n 20 of Fig. 5 



Fig. 7 is a flow chart of a pre-f iltering procedure 
performed by the software of Fig. 2. 

Figs. 8A, 8B and 8C are state graphs representing 
nodes of the lexical tree of Fig. 5. 
25 Figs. 9 and 10 are charts of scores corresponding to 

the states of the state graphs of Figs. 8A, 8B and 8C. 

Fig. 11 is a flow chart of a procedure for 
processing nodes of a lexical tree. 

Fig. 12 is a flow chart of a speech recognition 
3 0 procedure . 
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Figs. 13A and 13B are block diagrams of a complete 
dictation vocabulary that does (Fig. 13B) and does not (Fig. 
13A) include word fragments. 

Figs. 14A and 14B are block diagrams of a complete 
5 dictation vocabulary for the French language that does (Fig. 
14B) and does not (Fig. 14A) include word fragments. 

Fig. 14C is a block diagram of the word fragment 
component of the vocabulary of Fig. 14B. 

Figs. ISA and 15B are block diagrams of a complete 
10 dictation vocabulary for the U.S. English language that does 
(Fig. 15B) and does not (Fig. ISA) include word fragments. 

Fig. ISC is a block diagram of the word fragment 
component of the vocabulary of Fig. ISB. 
O Fig. 16A is a flow chart of a procedure for 

W IS generating word fragments that will be used in a speech 
^2 recognition system. 

y Fig.l6B is a flow chart of a procedure for using 

y generated word fragments in the speech recognition system. 

Fig. 17 is a flow chart of a procedure for 
P 20 determining word fragments as part of the procedure of Fig. 
16. 

SI Fig. 18 is a flow chart of a procedure for 

'^1 estimating word fragments as part of the procedure of Fig. 

17. 

25 Fig. 19 is a flow chart of a procedure for 

estimating affixes as part of the procedure of Fig. 18. 

Fig. 2 0 is a flow chart of a procedure for finding 

useful spelling rules associated with the affixes as part of 

the procedure of Fig. 19. 
3 0 Fig. 21 is a flow chart of a procedure for 

estimating and tagging roots as part of the procedure of 

Fig. 18. 
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5 a Butficientiy ^^^^ utterance may 

pause haviny mi lliseconds) • 

^£or example, 160-2=0 mill 
nolude one or more °J^„^„„,„ts of the speech 

Bia 2 illustrates Co v discussion, the 

.ti!n software 160. ^or ease of 6 ^^^^ 
recognition B ^^^^^ the ^^^^^ 

following aescrip . ^.^^ '„,Uy causes the 

out operations to ach ,„^„„ent actually 

. .hould .e the specified manner, 

processor 130 to operate ^^^^^^^^^ ,00 

initially, a fron ^„^„d card 



S3 
W 

o 

, ri 



into frames of parameters 210 that represent the frequency 
content of an utterance. Each frame includes 24 parameters 
and represents a short portion (for example, 10 
milliseconds) of the utterance. 
5 As shown in Fig. 3, the front end processing module 

200 produces a frame from digital samples according to a 
procedure 300. The module first produces a frequency domain 
representation X(f) of the portion of the utterance by 
performing a Fast Fourier Transform (FFT) on the digital 
10 samples (step 305). Next, the module determines log(X(f))^ 
(step 310) . The module then performs frequency warping 
(step 315) and a filter bank analysis (step 320) to achieve 
speaker normalization. See S. Wegmann et al . , "Speaker 
^3 Normalization on Conversational Speech, " Proc . 1996 ICASSP , 

iy 15 pp. I. 33 9- I. 341, which is incorporated by reference. 
^3 From the normalized results, the module performs 

Q cepstral analysis to produce twelve cepstral parameters 

(step 325) . The module generates the cepstral parameters by 
performing an inverse cosine transformation on the 
0 20 logarithms of the frequency parameters. Cepstral parameters 
and cepstral differences have been found to emphasize 
information important to speech recognition more effectively 
than do the frequency parameters. After performing channel 
normalization of the cepstral parameters (step 330) , the 
25 module produces twelve cepstral differences (that is, the 
differences between cepstral parameters in successive 
frames) (step 335) and twelve cepstral second differences 
(that is, the differences between cepstral differences in 
successive frames) (step 340) . Finally, the module performs 
3 0 an IMELDA linear combination transformation to select the 

twenty four most useful parameters from the twelve cepstral 
parameters, the twelve cepstral differences, and the twelve 
cepstral second differences (step 345) . 
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<recognition result> ::= Select <word>, 

where 

<word> : : = [PRW' [PRW^ [PRW^ . . PRW"] ] ] | 
[PRW' [PRW\ . PRW"] ] I . . PRW^, 

5 "PRW^" is the previously-recognized word i, 

[ ] means optional, 

< > means a rule, 

I means an OR function, and 

means "is defined as" or "is". 

10 As illustrated in Figs. 4A and 4B, this notation indicates 

that "select" may be followed by any ordered sequence of 

previously-recognized words. This grammar does not permit 

optional or alternate words. In some instances, the grammar 

O may be modified to permit optional words (for example, an 

15 optional "and" to permit "four score and seven" or "four 

score seven") or alternate words or phrases (for example, 

a==t 

"four score and seven" or "eighty seven"), 
vj One constraint grammar 22 5 that may be used by the 

Q speech recognition software 160 is a large vocabulary 

f=t 20 dictation grammar. The large vocabulary dictation grammar, 
^.P which also may be referred to as a language model, 

,j identifies words included in the active vocabulary 230, 

which is the vocabulary of words known to the software. The 
large vocabulary dictation grammar also indicates the 

2 5 frequency with which words occur. The language model may be 

a unigram model that indicates the frequency with which a 
word occurs independently of context, or a bigram model that 
indicates the frequency with which a word occurs in the 
context of a preceding word. For example, a bigram model 

3 0 may indicate that a noun or adjective is more likely to 

follow the word "the" than is a verb or preposition. Other 
constraint grammars 225 include an in-line dictation macros 
grammar for dictation commands, such as "CAP" to capitalize 
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;::n:::::^r"by 
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files may be associated with 
one or more voc^ulary £x 
each user. The vocabulary tor the user, 

pronunciations and ^^^uage model ^^^^^ ^^^^^^^^^^ 

mctatlon and command ^^^'^^^^'l^,,,^,,^. and memory use, 
nies to optimize ^^^f^^l^^^J^ ,,,e under the size limit 
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The acoustic models 235 
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The acoustic models ^3 functions 
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(,.PDFS"). For example, node 
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ab'c ^l^w,^(V'k-''k^ ■ 



a 

W 



C3 
W 



Q 20 



V. ,r is a mixture weight, 
25 where each w^ is a n 

5> = 1 / 



, , the probability density function 
is a mean — ^ f^f ^v.rlance matrix for the PDF 

("PDF") N„ and c, is tn 
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expands to a 9^°^^ ^ ^^ence node 512 

,i,, .hich words start^^^^^ ^^^^ ^°°Ut a P--^ ^^^^ 
silence also may be ' For example. 

Each node m the g ^^^^ ^^^^^ . For 
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in the portion 600 of the lexical tree 500 illustrated in 
Fig. 6, a node 610 corresponds to all words in the 
vocabulary that start with the phoneme "H" . Together, the 
nodes in the group 510 include representations of the 
5 starting phoneme of every word in the vocabulary. 

The lexical tree continues to expand until it 
reaches leaf nodes 515 that represent the actual words of 
the vocabulary. For example, as indicated by the square 
marker, leaf node 615 of Fig. 6 corresponds to the word 
10 "healing" . An internal node of the tree also may represent 
a word of the vocabulary. For example, the node 52 0 might 
represent a particular vocabulary word in addition to 
representing the first two phonemes of other vocabulary 
words. Similarly, the leaf node 620 of Fig. 6 corresponds 
15 to the words "heal" and "heel" while also corresponding to 
the first three phonemes of the words "heals", "heels" and 
"healing". Node 620 also illustrates that, since multiple 
words may have the same phonetic spelling, a leaf node may 
correspond to more than one word. As illustrated in Fig. 6, 
2 0 leaf nodes may appear at different levels within the lexical 
tree. Leaf nodes also may correspond to commands. For 
example, a leaf node may correspond to the word "select" and 
=:Q to the command "SELECT". As noted above, commands may be 

associated with particular constraint grammars 225. 
25 Operation of the pre-f iltering procedure 240 is 

illustrated in Fig. 7. The pre-f iltering procedure begins 
by retrieving the next frame of parameters for an utterance 
(step 700) . Immediately after initialization, the next 
frame will be the first frame for the utterance. 
30 Thereafter, the next frame will be the frame following the 

last frame that was processed by the pre-f iltering procedure 
when the pre- filtering procedure was last called. The pre- 
filtering procedure does not reinitialize the lexical tree 
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between requests for list of words. Accordingly, the state 
of the lexical tree when a list of words is requested 
corresponds to the state of the lexical tree after a 
previous list of words was returned. 

After retrieving a frame of data, the pre-f iltering 
procedure 24 0 finds an active node in the tree with no 
unprocessed active successors (step 705) . Successors of a 
node also may be referred to as subnodes of the node. When 
the lexical tree is initialized, the silence node 512 is the 
only active node. 

Next, the pre-f iltering procedure 240 processes the 
current node (step 710) according to a node-processing 
procedure 1100 that is discussed below with reference to 
Fig. 11. The node-processing procedure determines whether 
the node should spawn additional active nodes and whether 
the node should be rendered inactive. If the node is a leaf 
node, the node -processing procedure also determines whether 
the word corresponding to the node should be added to a word 
list for a time associated with the node. 

After processing the node (step 710) , the pre- 
filtering procedure 240 determines whether the node is the 
highest node in the tree (that is, the root node) (step 
715) . If the node is not the highest node, then the pre- 
filtering procedure 240 goes to the next node having no 
unprocessed active subnodes (step 72 0) and processes that 
node (step 710) . When searching for the next node to 
process, the pre-f iltering procedure 240 considers inactive 
nodes having active subnodes or active siblings. 

If the processed node is the highest active node 
(step 715) , then the pre-f iltering procedure 240 processes 
the silence node 512 (step 725). In general, the silence 
node is processed by comparing a frame to a model for 
silence and adding the resulting score to the minimum of the 



current score for the silence node and the score for the 
root node 505. 

Next, the pre-f iltering procedure 240 reseeds the 
lexical tree 500 (step 730) . The pre-f iltering procedure 
240 reseeds the tree whenever the silence node 512 is active 
or a word was produced by a leaf node of the lexical tree, 
regardless of whether the word was added to the list of 
words. The pre-f iltering procedure reseeds the tree by 
replacing the score for the root node 505 with the minimum 
of the score for the silence node 512 and the scores for any 
words produced by leaf nodes of the lexical tree for the 
current frame. If the silence node is inactive and no leaf 
node has produced a word, then the pre-f iltering procedure 
replaces the score for the root node 505 with a bad score 
(that is, a score having a value larger than a pruning 
threshold) . 

Next, the pre-f iltering procedure 240 determines 
whether more words may be added to the word list for the 
requested time (step 735) . If there are no active nodes in 
the lexical tree 500 corresponding to speech that started 
at, before, or slightly after the start time for which the 
list was requested, and if the last frame to be processed 
corresponds to a time that is slightly after the start time 
for which the list was requested, then no more words may be 
added to the word list. A word produced by the lexical tree 
is added to the list of words corresponding to the start 
time of the word and to lists of words corresponding to 
times that precede and follow the start time of the word. 
It is for this reason that the pre-f iltering procedure waits 
until there are no active nodes in the tree corresponding to 
speech that started slightly after the start time for the 
list of words. If more words may be added, then the pre- 



filtering procedure 240 retrieves the next frame of 
parameters (step 700) and repeats the steps discussed above. 

If words cannot be added to the word list (step 
735) , then the pre-f iltering procedure 240 returns the word 
list (step 740) to the recognizer 215. If the word list 
includes more than a predefined number of words, then the 
pre-f iltering procedure removes words from the list prior to 
returning the list. The pre-f iltering procedure removes the 
words that are least likely to correspond to the user's 
speech and removes enough words to reduce the number of 
words on the list to the predefined number. The procedure 
240 also deletes any lists of words for times prior to the 
requested start time. 

Each node of the lexical tree 500 (Fig. 5) 
represents a sequence of states for a particular phoneme. 
For example. Fig. 8A illustrates a node 800 that includes a 
first state 805, a second state 810, and a third state 815. 
A comparison with a frame of parameters may cause the score 
in a particular state to remain in the state (through a path 
820) . A score remains in the state when the score, after 
being adjusted based on a comparison with a model for the 
state, is better than a score passed from a preceding state 
or node, or when no score is passed from a preceding state 
or node. The comparison also may cause the score to be 
passed to a subsequent state through a path 825. A score is 
passed to a subsequent state when the score, after being 
adjusted based on a comparison with a model for the 
subsequent state, is better than the score in the subsequent 
state, or when no score is associated with the subsequent 
state. The score for the third state 815 may be passed to 
one or more subsequent nodes through a path 830. 

Referring to Fig. 8B, the node 512 that corresponds 
to silence is represented by a single state 840. Each 



comparison with a frame of parameters may cause a score in 
the node to remain in the state 84 0 (through the path 84 5) 
and also may cause the score to be passed to the root node 
505 through a path 850. 
5 Referring to Fig. 8C, the root node 505 is 

represented by a single state 860. Comparison with a frame 
causes the score in the node to be passed to one or more 
subsequent nodes (including the silence node 512) through a 
path 865. 

10 Each state of a node may be represented by four 

values: a score, a starting time, a leaving penalty, and a 
staying penalty. The score represents the likelihood that a 
series of frames has placed the lexical tree in the state 
O (that is, the probability that the series of frames 

ui 15 corresponds to the word or portion of a word to which the 
^^3 state corresponds) . The scores are maintained as negative 

a logarithmic values. 

The starting time identifies the hypothesized time 
at which the user began to speak the word or words 
C3 20 represented by the state. In particular, the starting time 
identifies the time at which the score associated with the 

Q 

state entered the lexical tree (that is, the time at which 
the score was passed from the state 84 0 along the path 850) . 
The leaving and staying penalties are fixed values 
25 associated with the state. The staying penalty is added to 
any score that stays in the state. The staying penalty is 
related inversely to the length of the sound represented by 
the state and to the length of the phoneme represented by 
the node to which the state belongs. For example, the 
30 staying penalty could be proportional to -log(l - l/d^^g) , 
where d^^g is the average duration, in frames, of the sound 
represented by the state. Thus, the staying penalty has a 
relatively large value when the sound corresponding to the 
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Fig. 9 provides a simplified example of how scores 
propagate through the lexical tree 500. Before the first 
frame is retrieved (row 900) , state 840 (which corresponds 
to silence) has a score of 0 and no other nodes are active, 
5 The score of 0 means that there is a one hundred percent 
probability that the system is starting from silence. 

After the first frame is retrieved (row 905) , the 
score for the state 840 (S;^^^) is set equal to the acoustic 
score (A;^i) resulting from an acoustic match of the first 
10 frame with an acoustic model corresponding to the state 840 
(that is, the acoustic model for silence) . Thus, the score 
for the state 840 (S^i) is set equal to the likelihood that 
the first frame corresponds to silence. 

Retrieval of the first frame also causes the state 



15 805 to become an active state. Assuming that the node 800 

• 

corresponds to a phoneme that starts a word, the score for 
Q the state 805 (S^i) is set equal to the acoustic score (AgJ 

^2 resulting from an acoustic match of the first frame with the 

'J^ acoustic model corresponding to the state 805. Thus, the 

P 20 score for the state 805 (SgJ is set equal to the likelihood 

that the first frame corresponds to the state 805. The 

1;— t 

%= Starting time for the state 805 is set equal to the time 

associated with the first frame. This value for the 
starting time indicates that the score at state 805 
25 represents a word that started at a time corresponding to 

the first frame. The starting time moves with the score as 
* the score propagates through the lexical tree 500. 

After the second frame is retrieved (row 910) , the 
score for the state 84 0 (S;^2) is set equal to the sum of the 
3 0 previous score for the state (S^^) and the acoustic score 
(A;^2) resulting from an acoustic match of the second frame 
with the acoustic model for silence: 
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^A2 ^Al ^A2 ^Al ^A2 ' 

As noted above, each of the scores corresponds to a negative 
logarithmic probability. Accordingly, adding scores 
together corresponds to multiplying the probabilities. 
Thus, the score for the state 840 (S;^2) equals the 
likelihood that both of the first and second frames 
correspond to silence. This process is repeated for 
subsequent frames (for example, lines 915 and 920) so that 
the score for the state 840 at a frame "n" (S^n) equals: 



^An ~ ^An-l ^An ~ ^ A 



n 

A 

Am ' 



lO This expression assumes that the silence node 512 is not 

W 10 reseeded from the root node 505. If reseeding occurs at a 

P frame n, then the value of S^n-i would be replaced by the 

iU score in the root node 505 for the frame n-1 . 

'"'4 

f-^- After the second frame is retrieved, the score for 

s the state 805 (3^2) is set equal to: 

q 

2 15 where is the acoustic score resulting from an acoustic 

n match of the second frame with the acoustic model 



corresponding to state 805 and stayg is the staying penalty 
for state 805. The score for state 805 (832) corresponds to 
the more likely of two alternatives: (1) the first frame 

2 0 was silence and the second frame was the sound represented 
by the state 805 or (2) both of the first and second frames 
were the sound represented by the state 805. The first 
alternative corresponds to a transition from state 840 to 
state 805 along the path 850. The second alternative 

25 corresponds to a transition from state 805 back to state 805 
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along path 820. When the first alternative is the more 
likely, the starting time corresponding to the first frame 
that was stored previously for the state 805 is replaced by 
a value corresponding to the second frame. This value 
indicates that the score at state 805 represents a word that 
started with the second frame. 

After the second frame is retrieved, the state 810 
becomes an active state. The score for the state 810 (Sc2) 
is set equal to: 

where A^^ is the acoustic score resulting from an acoustic 
match of the second frame with the acoustic model 
corresponding to state 810 and leaveg is the leaving penalty 
for the state 805. Similarly, leavec and leave^ are leaving 
penalties for, respectively, states 810 and 815. The sum of 
language model components of leaveg, leavec and leave^ 
represents the language model score for the phoneme 
represented by the node 800. 

The methodology for determining state scores for 
states other than the silence state can be expressed more 
generally as: 

j =min (5^^^ + stay^, S-_^j^^ + leavej_^) -^A^^. 

for i greater than zero (where i equals zero corresponds to 
silence) , and with the boundary condition that the score for 
an inactive state equals infinity or some sufficiently large 
value. The starting time for the state may be represented 
as : 

t^j=t^^j_^ for S^^j_^+stay^<Si-l, j-l+leavej_^, 

or 



for i and j greater than zero and with the boundary 
condition that the time value for a newly active state 
represents the frame at which the state became active. As 
previously noted, state scores for the silence state may be 
5 determined as: 

j 
m=l 

with the boundary condition that Sq^o equals zero. An even 
more general form, in which the scores are expressed as 
functions of the various parameters, is illustrated in 
!q Fig. 10. 

10 Referring to Fig. 11, a node may be processed 

according to a node-processing procedure 1100. Initially, 
W the node-processing procedure updates the scores and time 

values for each state of the node (step 1105) . The node- 
processing procedure updates the scores and time values by 
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15 generating acoustic scores and using the equations discussed 
above . 

S When the last state of the node was active prior to 

% updating the scores for the node, the node -processing 

procedure 1100 uses the score for the last state to generate 
20 scores for any inactive subnodes of the node. If the 

generated score for a subnode does not exceed a pruning 
threshold, then the node-processing procedure activates that 
subnode and provides the subnode with the generated score. 

Next, the node-processing procedure 1100 determines 
2 5 whether the score of any state of the node exceeds the 

pruning threshold (step 1110) . When a score exceeds the 
pruning threshold, the likelihood that the word represented 
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pre-filtering , language model score 
to a crude acoustic ^^P*"""" ^ ^^^^ i^ used, 

indicative of the ;;::;J„:,,r 21S .ay eliminate 

independently of context . ^^^^ ^^^^ ^ constraint grammar 

' Z rrt:::^: "vP^thesis, , hut does not comply 

„ith the the recognizer 21S operates 

referring to Fig. ^^^^^^ ^^^^^ processing, 

according to a procedure 12U 

initializes the iexioa-t 
5 the recognizer recognizer 215 then 

described above (step 12u , . determines 
retrieves a frame of parame te - P^^^^^^^^ 
whether there are ^VPOth ;-;^,„,^, corresponds to silence 
„,,p 1,15) . The £"-'J"J/^ considered for the 

30 so that there are no hypotheses 

first frame. considered for the frame 

If hypotheses need to be c ^.^^^ 

(step 1215) , then the recognizer 213 goes 
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hypothesis (step 1220) . The recognizer then compares the 
frame to acoustic models 235 for the last word o£ ^^e 
hypothesis (step 1225) and, based on the comparrson, updates 
a score associated with the hypothesis (step 1230) . 

;.£ter updating the score (step 1230) , the recognrzer 
215 determines whether the user was likely to have spoken 
the word or words corresponding to the hypothesis (step 
1235) The recognizer 215 makes this determination by 
comparing the current score for the hypothesis to a 
compar g exceeds the threshold value, 

10 threshold value. It the score e*^ 

then the recognizer 215 determines that the hypothesis rs 
too ullikely I merit further consideration and deletes the 

hypothesis (step 1240) . ^ ^ 

-^^y determines that the word or 

If the recognizer 2lb aeuexiuxuco 

words corresponding to the hypothesis were ^^^^l/ - ^^^^^ 
been spoken by the user, then the recognizer 215 ^-^"^ 
O whether the last word of the hypothesis is ending (step 

'rf i'r5) The recognizer 215 determines that a word is ending 

when the frame corresponds to the last component of the 
model for the word. If the recognizer 215 ^^l^-^^^lZ.. 
„ord is ending (step 1245) , the recognizer sets a flag 
indicates that the next frame may correspond to the 
3 beginning of a word (step 1250). 

S If there are additional hypotheses to be considered 

25 for the frame (step 1255) , then the recognizer 215 selects 
the next hypothesis (step 1260) and repeats the co„^arison 
(step 1225) and other steps. If there are no more 
h^^otheses to be considered for the frame (step 1255) then 

^ . V, t-v,^iv t-Vi(a-re are more frames to 

the recognizer determines whether there are mo 
cne reuuyii , 4.^^ lofiq^ The recognizer 

30 be considered for the utterance (step 1265). Tne ^ ^ 

. ^h.i- there are more frames to be considered 
215 determines that tnere axe 

„hen two conditions are met. First, more frames must be 
available. Second, the best scoring node for the current 
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rr.r.r-^ of a predetermined number of 
frame or for one or more of pr ^^^^^ 

J' nreceding frames must have oeen 

.mmedxately precede g utterance has ended when 

and to. a predetermined nu^er of consecutive ^ 
,,,..aing £ra-s, . ^^^^^^ ^^^^^^^^^^ 

If there ar 

and the flag ind.catrng ^''-^J ^"jj considered 
^ -n= t-hpre were no hypotneseb uv^ ^ 
,.tep 1270) , or If there ^^,„^„,,er 21S requests 

£or the frame (step 1215) . tn ^^^^ 
from the pre-filtering procedure ^40 a 1 st 

^ ^ with the next frame (step 12/5). 
may start with the n ^^^^ 

„pon *^^,\^;\:,„i.er 215 uses the list of 

''T^::^^^^-- - --"^ hypothesis for 

words to create hyp ^^^^ ,^ -^^^t 

..ich a word has ^^^^ ,,,,, ,,,ing a list 

Of words has ~ll\ZZ.er 21B modifies the Ust 
word to a hypothesis, the recog ^.^^ ,eore 

,,ore (SJ for the word to produce a modxf.ed 
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. T is a language model score that represents the 
where L, is a lang g ^^^^ includes the 

freauency with which the pair or w 

worl and the immediately preceding word .n the 

hypothesis are used ^^^^^^ ^^J^Z' ^^rres^on,. to 

model score ^^^^f ^^JfJ /rd is used in speech, 
the frequency wrth whrch the list ^^^^ 
without reference to contex ^ T^a - g ^ ^^^^^^^ 

the modified list score '° ^^^^"'^ result is 

^ result to a threshold value, 

compares the result recognizer maintains 

less than the threshold value, 
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hhP recoqnizer determines that 

. active constraint grammars 225 and 

hypotheses to ^^^^/^ corresponds to no active 

abandons any hypothesis that P ^^.^i^ves the 

constraint grammar. The recognizer 215 then ret 
constra y repeats the procedure, 

next frame (step 1210) ana p process, then 

If there are no more speech frames P 
nLr 215 provides the most likely hypotheses to 
the recognizer 215 pro recognition candidates 

the control/interface module 220 as recog 

^^^^P ^^^^^ ■ . Pic 2 the control/interface 

Referring again to Fig. ^, "-''^ 

Kererxi y the speech recognition 

J 1 oon rnntrols operation ot tne spee<-i 
module 220 controls p .^^erface to other software or 

software 160 and provides an interface 

V- The control /interface module 220 receiv 
to the user. The cont ^ utterance from the 

of recognition candidates for eacn u 

"215 Recognition candidates may correspond to 
^^'^'rd tex speech recognition commands, or external 
dictated text, speecn y ^ecoanition candidate 

corresponds to d.cta ^^^^ .^pUcation, such as a 

.20 provrdes the ,,aule 220 also ™ay 

r^rrrest- r n red^nition candidate to the user 
display the best ,„t^-rfe,ce When the best-scoring 

i through a graphical user interface ,,,j,,e 
recognition candidate is a command, the / 

.odul .20 i^ple^ents the ^^J:;—;;;! speech 

n / • ^-or-fare module may control operation 
control /interface moau speech recognition 

■Ft-war-P 160 in response to speecn j.c ^ 
recognition software ibu f aoDropriate 

fr,^ward other commands to the appropxx 
0 commands and may forward ocne 

software. , controls the 

The control/interface module 220 also con 

,cti.e Jota^lary .30, acoustic models 235, and constraint 
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v-e that are used by the recognizer 215. For 
ZZr: Ten - speec. ^..on so.w.. «0 ..... 

• •*-v, = T->3-rhi rular application (tor 
used in conjunction with a particular yy ^ ^on 

f^ word) the control/interface module 220 
pvamole, Microsoft Word) , cne ^-'Jiiv- 

updlt . the active vocabulary 230 to Include co.™and words 
::fociated Wit. that application and activates constraint 

vo 99R associated with the application. 
^"""^ other ;rctions provided the control/interface 
module 220 include an enrollment program, a vocabulary 
Tustomlzer, and a vocabulary — r. The enrollment 
progran, collects acoustic information fro. a us r and trains 
or adapts a user's models based on that ^ 
vocabulary customi.er optimi.es the 
specific topic by scanning user supplied text The 
Tcabularv manager is a developer tool whic .s use 
hrowse and manipulate vocabularies, S~ and macros^ 
Each function of the control /interface module 220 may be 
Tm^med as an executable program that is separate from 
d the main speech recognition software. ^,^,^tion 

,0 Referring also to Fig. 13A, a complete drctation 

-1 the active vocabulary 230 plus a 

vocabulary 1300 includes the active 
O backup dictionary 245. The backup dictionary 245 may 

"crude user-specific backup vocabulary "^ds and system- 

„.de backup vocabulary words^ Thirr: a^ adred while 

vocabulary words include words that a user 
usinq the speech recognition software. In one 
using tne p ^ , ^ ^ limit of 64,000 user-specific 

implementation, there is a limit or , , files 

^^^o ar-P c,tored in vocabulary tiies 
words. User-specific words are storea 

for the user and for the dictation topic used during their 
for the user dictionary 
addition, and are available as part oi 

or the dictation topic regardless of the user, and to the 
user regardless of the dictation topic. For e-mp e i a 
u er using a medical topic adds the word "ganglion" to the 
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dictation vocabulary 1300, any other user of the medical 
topic will have immediate access to the word "ganglion". In 
addition, the word will be written into the user-specific 
backup vocabulary, so that, if the user later says 
"ganglion" while using a legal topic, the word "ganglion" 
will be available. The system-wide backup vocabulary words 
include all other non-active words known to the system. 

The control/interface module 220 implements error 
correction procedures of the speech recognition software 
160 When the speech recognition system 100 makes a 
recognition error, the user may invoke an appropriate 
correction command to remedy the error. During error 
correction, word searches of the backup dictionary 245 start 
with the user-specific backup dictionary and then check the 
system-wide backup dictionary. The backup dictionary 245 
also is searched when there are new words in text that a 

user has typed. 

in general, the backup dictionary 245 includes 
substantially more words than are included in the active 
vocabulary 230. For example, when the active vocabulary 230 
has 60,000 or so entries, the backup dictionary 245 may have 
roughly 190,000 entries. The active vocabulary 230 is a 
dynamic vocabulary in that entries may be added or 
subtracted from the active vocabulary over time. For 
3 example, when the user indicates that an error has been made 
and the control/interface module 220 uses the backup 
dictionary 245 to correct the error, a new word from the 
backup dictionary 245 may be added to the active vocabulary 
230 to reduce the likelihood that the error will be 

0 repeated. , 

Referring also to Fig. 13B, the effective size of 
the active vocabulary 230 may be increased by dividing words 
into constituent components or word fragments 1315. Word 



fragments include prefixes and suffixes which are 
collectively known as affixes. A prefix is any word 
fragment that precedes another word fragment or word and 
therefore may be represented as "prefix.". For example, a 
common prefix used in the English language is "re.". In 
contrast to a prefix, a suffix is any word fragment that 
follows another word fragment or word. A suffix may be 
represented as ".suffix". For example, a common suffix used 
in the English language is "+ed". 

word fragments also include roots that can join with 
a prefix or a suffix. A root may be a true root, in which 
case it is not a complete word when used alone and therefore 
may be represented as ".root.". An example of a true root 
is ".crimin." which may form the word "decriminalize" when 
preceded by prefix -de." and followed by the suffixes, in 
succession, ".al" and ".ize" . A root also may be a word 
root that is a word when used alone. For example, the word 
root "look" may be used as the word "look" or may be joined 
with other word fragments to form another word such as 
" looked" . 

word fragments may also include separators, such as 
the "s" used in the German language to join words together 
or the hyphen which is frequently used in the English 
language, and infixes, which are word fragments positioned 
in the interior of a word in such languages as Hungarian 
Mon-Khmer. For example, in the language Tagalog "tawa" 
means "laugh," but "t -um- awa" means "to laugh". Thus, 
-um- could represent an infix in the Tagalog language. As 
another example, in the Thai language, "pak" means "rest", 
however "p -n- ak" means "a support" such as a chair. Thus, 
-n- could represent an infix in the Thai language. Infixes 
are rare in the English language. An example of an informal 



infix used in English might be -bloody- as in "fan -bloody- 

t3.st ic " 

Each word fragment 1315 is included as a separate 
entry in the active vocabulary 230 along with a set of 
5 active words 1320 that typically includes a large portion of 
the previously active words 1312. The word fragments 1315 
increase the effective size of the active vocabulary to 
include words that may be in the backup dictionary 245, 
designated as splittable words 1325, that may be formed from 
10 combinations of word fragments with other word fragments or 
with active words. The words of the backup dictionary that 
may not be formed from these combinations are designated as 
non- splittable words 1330. 

When the recognizer 215 detects a word fragment 1315 
in an utterance, the system combines the word fragment with 
one or more other word fragments or words to produce a new 
Q word, and then searches the backup dictionary 245 or applies 

rules to confirm that the new word is a valid word. Thus, 
the speech recognition system 100 is able to tap into the 
backup dictionary 245 without requiring the recognizer to 
process words of the backup dictionary 245. By increasing 
S the effective size of the active vocabulary 230, without 

1 increasing its real size, the recognition coverage of the 

i speech recognition system 100 is expanded without 

25 significantly increasing the required memory capacity or 

processing time. 

The backup dictionary 245 may include words that are 
not recognized as a concatenation of a sequence that 
includes words and word fragments. For example, many proper 
30 names may be included in the backup dictionary. The name 

"Scarlotti" is one example of a proper name that may not be 
splittable. 
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Figs 14A and 15A show exanples of the size of the 
complete dictation vocabulary 1300 of one implementation 
for respectively, the French language and the U.S. version 
"he nglish language, without the use of word fragments. 

t his implementation, the complete Prench 

in cuxE, f words with about 

i_ 1 ,v-i7 i4nn includes about iau,uuu wui<-i 
vocabulary 1400 xnciu vocabulary 
40 000 of those words being m the Frencn ac 
03 By contrast, in this implementation, the complete U.S. 
230. By contr ^^.^^^ 15OO includes about 240,000 

Enqlish dictation vocabulary 150U ir 

T7r.rrl-ic^h activB vocabulary 230. 

'referring to Fig. l^B, word fragments may be used to 
. se the effective size of the French active vocabulary 

;r: :ie leering the co^lete dictation —-y UOO 

J ;:5rtive vocabulary 23U neax 

iftn 000 words and tne aciiivc v^v^ 
near about 180, OUU wox active vocabulary 

.0,000 elements. ^r^e^ and about 

r:„rFr: :: rd frigiir-o. such that the word 

; Ln s 40B total about S. of the total active vocabulary 
AS Shown, in this implementation, a -bstan lal 
on of the backup dictionary constitutes splittable 
'° ; l^fo that may be accessed from the active vocabulary 
words 1410 that constitutes non-splittable 

230, while a much smaller portio 

J. 111=; Referring also to Fig. 14C, the trenc 

e ts 140^ mdude mostly French true ^^ots 14.0 .t a. 

rFrlLTr^-ri^S constitute a small segment of the 

Pr-e-nch word fragments 1405. 

Frencn wo ^ word fragments also may be 

Referring to Fig. i^a, w'-'-l^ ^ 

the effective size of the U.S. English 

::rv:=y'^:o:'wbiie .eping the 

vocabulary 1500 near about 240,000 words and the 
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vocabulary 230 at 30.000 elements. The active vocabulary 
vocaDuxaxy Pnalish words 1502 and 

230 includes about 28,800 active U.S. Englxsh wo 
about 1,200 U.S. English word fragments 1505, such that 
ord ««.-ts 1505 total about « of the tota aot.ve 

vocabulary 230. ^ ^^^^^^Z^^^ t^l^^ 
dictionary constitutes splittable wor 

accessed from the active vocabulary 230, while a larger 
plrt L constitutes non-spUttable words 1515. -ferring 
Tso to Fig. 15C, the set of U.S. English word fragments 
.S05, in contrast to the Erench word ^^^^ 
includes quite a small percentage of ^"9^;* 
,S20 in this particular implementation, "^^^^ "^^' J^f 
,oots 1525 constitute the largest segment of U^S^ word 
fragments 1505. In this implementation, U.S^ ^'^f ^^.^^ 
suffixes 1530 and prefixes 1535 constitute about one third 
:f English word fragments 1505, a substantially 

larger portion than the true roots, m another -re recent 
ementation, the set of U.S. English word fragmens 505 
may be larger or may include a larger percentage of true 



roots 
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Generation_ofJio^ ^ 
;^-^7;~T6^word fragments may be 

generated in the speech recognition system 100 through a 
Lcedure 1600. This procedure 1600 is implemented 
!of line"- that is before the speech recognition software 
"Offline , cn procedure is performed by a 

160 is sold and installed. The procea y 

^v.=,n the soeech recognition system 100. 
3ystem ,_,„.e ISOO, word fragments 

•l,-nc^ Thpn using the word 
17m are identified step 1605). Then, usi y 
1315 are laen^ ^^v>,navv 230 and a new backup 

fragments 1315, a new active vocabulary 230 

dictionary 245 are generated (step 1610) . 



In one implementation, the new active vocabulary is 
constrained to be the same size as the old active 
vocabulary, such that the word fragments 1315 replace the 
least frequently used words of the active vocabulary. For 
5 example, in the implementation shown in Fig. 14B, the new 
French active vocabulary 230 has the same total number of 
elements (that is, 40,000) as the old French active 
vocabulary shown in Fig. 14A, but now includes word 
fragments 1405 and a corresponding reduction in the number 

10 of active words. 

In other implementations, the new active vocabulary 
may be larger or smaller in size than the old active 
vocabulary. 

After the new backup dictionary and the new active 
15 vocabulary are built, the language model is created (step 
1615) . This permits the speech recognition system 100 to 
use the created language model to recognize both word 
fragments and words, and, in particular, to recognize word 
fragments in the context of surrounding word fragments and 
20 words. 

TTciin q Word Fr;=)aments (procedure 1650) 

Referring also to Fig. 16B, once the off-line 
procedure 1600 is completed, the speech recognition system 
100 is ready to perform speech recognition using the updated 

25 language model and the new active vocabulary according to an 
"on-line" procedure 1650. To this end, the system 100 
performs speech recognition on an utterance to generate a 
list of candidates (step 1655) . During speech recognition, 
the recognizer 215 operates according to the procedure 1200 

30 shown in Fig. 12, with the exception that the recognizer 215 
is able to detect word fragments 1315 as well as words. 
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After the recognizer 215 generates the list of 
candidates for an utterance, the system 100 performs a post- 
recognition procedure (step 1660) . During the post- 
recognition procedure, the system 100 combines any word 
5 fragments 1315 that may be included in a recognition 

candidate to form a word and loo.s up the word .n the backup 
dictionary 245 or applies rules to verify that .t .s a val.d 
word, candidates that include word fragments which form 
invalid words are discarded. 

in the following discussing, the off-line procedure 
1600 is explained with respect to Figs. 17-28. Following 
that explanation, the on-line procedure 1650 is explained 
with reference to Figs. 12, 29 and 30. 
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TH^ntifv Fr^ qn^^T^t-.q (step 1605 L 

f-gments 1315 that will 

be included in the new active vocabulary 230 are identified 
according to a computer- implemented procedure 1605. First, 
3 and as discussed below with reference to Fig. 18, the 

n ^ -;r.-ih-i^T cif^t of word fragments (step 

computer postulates an initial set or wo y 

3 20 1700) . AS shown in Figs. 14C and 15C and discussed above, 
word fragments 1315 may include affixes (that is, prefixes 
and suffixes) , roots (both true roots and word roots) , 
separators (useful, for instance, in Germanic languages), 
and infixes. However, the described implementation 
postulates only roots, prefixes, and suffixes^ Hach pre ix 
or suffix includes a list of one or more spelling rules that 
define how the prefix or suffix may be combined with a root 
or word to form another word. 

once the word fragments 1315 are postulated, the 
computer uses the word fragments 1315 to split ^-^^^J 
the backup dictionary 245 (step 1705) . For each word of the 
baCup dictionary, the computer determines whether the word 
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. ,ero or more prefixes, 
^3^3,ted using a sequence ^ ^^^^ence 

be generate ^^^.^^ ^^'^'Je spelling rules 

a root or a word r employing the sp 

more suffixes, ana f ^^^^ 
of zero or mor ^^.^^^ ,xes 

associated w.th the p ^^^^^^^^ ^^^,3, ^^^^^^ 

implementations may P ^^rds from 

include multipJ-e ^e^arilY monitor 

sequence may mci ^e^essariiy 

V. i;,rv The computer a ^,,,iarY 230 may be spi 

vocabulary. ^""^"^ the old active 

whether words from ^^^^^ the oi 

v.- ctaae because most oi .^^ive vocabulary, 

''''' r V will remain in the new acti ^he 
vocabulary wiU ^^.^^ ,ord fragm ,^ 

because the purpose o ^^^^^ ^^^^ not £ 

Kiii^irv IS to ^1^3 backup 

^^tive vocabulary ^^^^^^ ^^^^.^ m the 
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because t^^- ^" ^ . ^/ords - 

K,iiarv IS to ro-^" ^j-g backup 

-cabulary ^^^^^^ 

aoti« vocabulary, ^^^^ 

dictionary 245. identified and keeps ^ 

The computer then others (step r 

r ff ixes, v,hile discarding th j„g„ents may 

: tion o£ *e best or most ^^^f^^l^, ,„,„ents in 

rbaTon a "U*« - --"^Z r - 

. o£ the bacl^up dictionary, the ^^^^ j„gments 

„ords o£ the ^.^^ionary w ^eral, the 

«ords o£ the backup .^^^ criterion. » 9 

or some o^-^^J^^^^^^ ,3Ch -^3--/^:;:^= the 

on or set o£ X :r--- 

particular enter g^t^sfy 

^ '^^''""r:t, the computer ^^iit »ords o. the 

„hich the comP-e=^ „o,d J^,, 

.ackup di"^"-':^;^^ computer postulates word « ^ ^^^^^ 
begin an i««"°"' ,he backup dictionary 245 ^ 

„ot be BPli^ „i,i„al P-^"^;"°"^„ ,„spUt 

This is similar ^^^^^^^ only 

except that the computer 



words in the backup aictionary 245 and uses the word 

Talents kept in step 1707 as part o£ the act.ve 

fragments p fragments to 

-^""""iitTorrr: e "lictionar. .step i7.= , 

t e Ira .ra^^ents ha.in. scores that satis.. 

rLesLid re,uire.ent ^^'^ ^^^^ ^^rTT^^^. 
„,uire™ent initially may be se to reta ^^^^^ 
larae number o£ word fragments 1315. on tn 
threshold retirement initially may be set to reta.n a 
relatively small number o£ word fragments. 

If the computer has not performed a designated 
^=^^T^ 1725) the computer begins 
""Ter^it:::™ a ^ng *e';hreshold retirement more 
Sfficul to tlsfy (step 1730, , and postulating a new set 
difficult to in „ost implementations, the 

; of word fragments (1710) .mm f .„„ther 

^ o c: -J hoT-^t ions . anouiiej- 
computer P"-'^- ^ " computer ^y begin a next 

implementation, at step i/^^> ^=oier to 

i/eration by ,«.ing the threshold ^^^'^^^J^^^^ a 
satisfy The threshold requirement may be varied in 

al manner for example, first high, then low, and so 
0 .igzag — ■ ' threshold retirement 

rid'rurrrarely t.c.X. and could establish a core set 

°' "-'r:t: :e":::er of iterations have been 
, Td Lap 1725) , the computer splits up the baclcup 
,S P-£°™^^ fragments, but does not 

dictionary words using £„gments used (step 

include Short ^^'^ ^.Tr Z.s have more detailed acoustic 
1735) . Generally, longer ruu 

=nH for this reason, are more 

Tbe"; e::.:: by the speech recogni.er 

30 likely to be correcciy ^^n^nalv if a v/ord may be 

^Hr^r-hpr roots. Accordingly, j-j- ^ 
215 than are shorter roots ^^^^ 

Hv a lonq root and a first seu 
represented by a long 

other fragments, or a short root and a 
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more other fragments, the word is more likely to be 
recognized when it is represented using the long root. For 
example, the word "accordingly" may be represented as 
-accord +ing +ly" or "ac+ cord +ing +ly" , with the first 
5 representation being the one more likely to result in 
correct recognition. By excluding short roots, such as 
"cord", from the first splitting step, the computer ensures 
that words such as "accordingly" will be represented using 
the preferred longer roots. The computer then includes the 
10 short roots in the set of word fragments and splits up the 
backup dictionary words that were not split using only long 

roots (step 1740) . 

Next a stop list of undesirable word fragments 1315 
may optionally be created (step 1745) . This stop list may 
be based on human intervention. For example, a person 
familiar with the language being processed may exclude word 
fragments that do not make sense. If a stop list is 
1 created, the computer then splits up words in the backup 

i dictionary into word fragments 1315 without using word 

h 20 fragments from the stop list (step 1750) . 

H Posi-.ulate T^raaments (step 1700) 

1 Postulation of fragments (step 1700) will now be 

3 discussed with reference to Figs. 18-22. Referring to Fxg. 

18 the computer first postulates affixes (step 1800) . In 
25 English, affixes include prefixes and suffixes. In other 
languages, as discussed earlier, affixes may also include 
infixes and separators. The following discussion assumes 
that the affixes include only prefixes and suffixes. 
However, it should be clear that other types of affixes may 
30 be generated using similar techniques. As discussed xn 
detail below with reference to Fig. 19, the computer 
generates a set of prefixes and suffixes and associated 
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spelling rules that may be used in conjunction with words of 
the active vocabulary to form words of the backup 
dictionary. In particular, for each word of the active 
vocabulary, the computer finds similar words in the backup 
dictionary and identifies affixes and associated spelling 
rules that may be used to map from the active word to the 
backup word or words . 

Once the list of possible affixes is formed, the 
computer finds the word roots (that is, roots that are 
words) and true roots (that is, roots that are not words) 
that may be used in conjunction with the postulated prefixes 
and suffixes to form words of the backup dictionary (step 
1805) . In particular, for each backup word ending with a 
suffix or beginning with a prefix, the computer finds the 
root or roots that may be combined with the affixes to form 
the word. The computer then keeps the most useful of these 
roots . 

Finally, the computer postulates a new set of 
prefixes and suffixes using an active vocabulary 
supplemented with the roots kept (step 1810) . With the 
exception of the change to the active vocabulary, this step 
is identical to the original (step 1800) . 

Postulate Affixes (step 1800) 

Referring to Fig. 19, affixes are postulated 
according to a generalized procedure 1800. In the described 
implementation, the procedure 1800 is performed once for 
suffixes and once for prefixes. The discussion below 
references suffixes, but is equally applicable to prefixes. 

Initially, and as discussed in detail below with 
reference to Fig. 20, the computer finds all spelling rules 
for suffixes (step 1900) . A spelling rule indicates how a 
word of the active vocabulary must be modified to form a 



word of the backup dictionary. For example, a spelling rule 
for converting the active word "hop" to the backup word 
"hopping" might be to add "ping" to the end of "hop", an 
alternative rule for the same purpose would be to replace 
the "p" at the end of "hop" with "pping" . 

Once the suffix spelling rules are found, the 
computer groups the spelling rules together to completely 
define the suffixes (step 1905) . Each suffix has an 
associated set of spelling rules that, when joined with the 
suffix, defines usage of that suffix in words. For example, 
a set of rules for the "+ing" suffix might be:* 

-null +ing 

-null +ling 

-null +ping 

-1 +lling 

-p +pping 

-e +ing 

-k +king 
where "-null" indicates that nothing is removed from the 
root before adding the suffix and, for example, "-p" 
indicates that a "p" is removed from the root before adding 
the suffix. 

Once the suffixes are defined, the computer keeps 
the most useful spelling rules for each suffix (step 1910) . 
This determination may be based on the frequency with which 
a spelling rule may be used to form a word, the relative 
frequency of the words formed by the spelling rule, or any 
other useful criterion. For example, the computer may 
determine that the spelling rule -null, +ping is redundant 
to the spelling rule -p, +pping, and may drop the -null, 
+ping rule for that reason. In general, a rule that removes 
one or more characters from the end of an active word is 
more precise than a rule that does not, which provides for 
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more efficient processing. Furthermore, if the rule that 
removes one or more characters can be used to split the same 
set of backup words as a rule that does not remove any 
characters, then the rule that removes one or more 
characters is favored for the reason that it provides for 
more efficient processing. For example, only roots or words 
that end with "p" must be considered with the -p, +pping 
rule, while all roots or words must be considered with 
-null, +ping. 

Next, the computer retains the most useful suffixes 
(step 1915) . Identification of the most useful suffixes may 
be based on the frequency of suffix occurrence or any other 
useful criterion. In general, suffixes that map to more or 
more common backup words are favored over suffixes that map 
to fewer or less common backup words. To retain the most 
useful suffixes, the computer associates a score with each 
suffix and retains the suffixes having scores that satisfy a 
threshold requirement, or the best-scoring suffixes up to a 
designated maximum number. 

After the computer has retained the most useful 
suffixes, it repeats the procedure 1800 for prefixes. The 
procedure for prefixes is the same as the procedure for 
suffixes . 

Find Spelling Ru les (step 1900) 

Referring to Fig. 20, spelling rules are determined 
according to a procedure 1900. Spelling rules are used to 
form affixes, and are used by the speech recognition system 
100 during the post -recognition processing to combine word 
fragments with other word fragments. A set of spelling 
rules is associated with each affix. To combine an affix 
with a root to form a word, one of the spelling rules for 
the affix must be followed to determine the final spelling 



of the word. For example, for the suffix "+ing, " one 
associated spelling rule might be to add "ing" to the end of 
the root and another associated spelling rule might be to 
subtract "p" from the end of the root and to add "pping" to 
the end of the root . 

First, the computer reads the whole backup 
dictionary 245 into a data structure, such as a Patricia 
(Practical Algorithm to Retrieve Information coded in 
Alphanumeric) Tree, to facilitate searching through the 
backup dictionary 245 (step 2000) . A Patricia Tree is a 
compact representation of a tree in which all nodes with one 
child are merged with their parents. Such a data structure 
can efficiently store and search sets of data items such as 
word strings, where the strings may be substantially the 
same except for a small number of bit positions. 

The computer then indexes all words in the active 
vocabulary (step 2005) and retrieves the first word (word j, 
where j = 0) (step 2010) . After retrieving a word, the 
computer searches through the data structure for words 
having spellings that begin in the same way as the retrieved 
word (step 2 015) . 

If any similar words are found (step 2020) , the 
computer stores all spelling rules that may be used to 
convert the active word to the one or more similar sounding 
words (step 2 025) . For example, for the active word 
"gripe", the computer might find the similar words "griped", 
"griping", "gripes", and "griper". Spelling rules generated 
for this sort of words might include "-null, +r" ; "-e, +ed" ; 
"e-. +ing"' "-pe, +ping"; "-null, +s"; "-e, +es"; and "-e, 
+er" . In general, the computer may limit the number of 
spelling rules by limiting the number of rules that may be 
used to convert from the active word to a particular backup 
word, or the number of letters that may be removed from the 
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active word in making the conversion. Thus, in the example 
provided above, only two rules per word and two removed 
letters per rule are permitted. This eliminates potential 
rules, such as "-pe, .ped" and "-ipe, wiping". 

After storing the spelling rules for an active word 
(step 2025) , or determining that there are no similar 
sounding words for the active word (step 2020) , the computer 
determines if there are more active words to process (step 
2030) If so, the computer advances to the next word m the 
active vocabulary by incrementing j (step 2035) , retrieves 
that active word (step 2010) , and processes the word (steps 
2015-2025) . 

If there are no more active words, the computer 
outputs all stored spelling rules that meet a predefined 
criterion (step 2040) . The criterion may be based on a 
frequency of occurrence. For example, the spelling rule 
..-pp .ing" may occur so infrequently that it would be 
dropped, while the spelling rule "-e, .ed" would occur more 

often and would be kept. 

The procedure 1900 is described above with respect _ 
to suffixes. However, the procedure would be changed little 
when finding prefixes. For example, for prefixes, the 
backup dictionary would be searched for words that end 
(rather than begin) similarly to the chosen active word. 

For some languages, some aspects of pronunciation or 
spelling may need to be ignored during the procedure 1900 
For example, in German, stress in pronunciation may need to 
be ignored and in French, vowel quality may need to be 
ignored. Case may be ignored for all languages. 

n^t-^r-mine ^^-^^^ («^-^^ l^Q^^ ^ . ^ . ^ 

Referring to Fi^l8, 21, and 22, after postulating 

the affixes (step 1800) , the computer determines the word 



roots and true roots according to a procedure 1805. 
Initially, the computer finds all possible roots that can 
precede suffixes or follow prefixes to form words of the 
backup dictionary 245 (step 2100) . This step may ignore 
more complicated spelling rules, and may only include rules 
in which the affix is added to the root. 

For each root that is retrieved (step 2105) , the 
computer determines if the root is "sufficiently close" to 
an existing word, where "sufficiently close" might be 
defined to be that all of the letters of the root and the 
existing word, except the last few or the first few, are 
identical (step 2110) . If the root is sufficiently close to 
an existing word, then the computer tags that root as a word 
root (step 2115) . If , on the other hand, the root is not 
sufficiently close to an existing word, then the computer 
tags that root as a true root (step 2120) . Once the roots 
are tagged, the computer determines if there are more roots 
to retrieve (step 2125) . If so, then the computer retrieves 
the next root (step 2130) and repeats steps 2110-2125. If 
there are no more roots to retrieve, the computer identifies 
and keeps the most useful new roots (step 2135) . The most 
useful new roots may be identified based on the frequency of 
occurrence of the roots in words of the backup dictionary or 
any other appropriate criterion. 

Find Possible Roots (step 2100) 

Referring to Fig. 22, the computer finds all 
possible roots that can precede suffixes or follow prefixes 
to form words according to the procedure 2100. Initially, 
the computer reads all words in the backup dictionary and 
all affixes into a searchable data structure (step 2200) . A 
first word is retrieved from the backup dictionary (step 
2205) and a first affix is retrieved from the set of affixes 



in the data structure based on similar spelling conODinations 
in the retrieved word (step 2210) . For example, if the 
retrieved backup dictionary word is "according", then a 
first affix that may be selected is "+ing" because this 
spelling combination occurs at the end of the word. 

Next, the computer determines a possible root by 
subtracting the retrieved suffix (or prefix) from the end 
(or beginning) of the retrieved word (step 2215) . The 
computer then stores the possible root for later processing 
(step 2220) . using the above, the retrieved suffix "-.mg" 
is subtracted from the retrieved backup word "according" to 
form the possible root "accord". 

After the root is stored, the computer determines if 
there are any other affixes that match (in spelling) any end 
or beginning of the retrieved backup word (step 2225) . If 
there are more affixes, the computer retrieves the next 
affix (step 2230) and forms a possible root based on this 
affix (step 2215) . If there are no more affixes associated 
with the retrieved backup word, the computer determines if 
there are anymore retrievable words in the backup dictionary 
(step 2235) . If so, the computer retrieves the next backup 
word (step 2240) and repeats steps 2205-2235. 

Postulate word Fraame r ^^ ^nr Unsp1 i f Words (step 1710) 

Referring to Figs. 17 and 23, after splitting the 
words in the backup dictionary (step 1705) , the computer 
postulates word fragments for unsplit words in the backup 
dictionary according to a procedure 1710. The computer 
initially finds word roots and true roots for unsplit words 
in the backup dictionary 245 (step 2300) . This is similar 
to the original procedure for identifying roots (step 1805) 
and can be understood with reference to Figs. 21 and 22. 
The only difference between the procedures is that is that 
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the computer now only checks unsplit words in the backup 
dictionary 245 whereas the computer previously checked all 
words in the backup dictionary 245. 

Once the word roots and true roots are determined 
for the unsplit words, the computer postulates the prefixes 
and suffixes that may be of use for splitting the unsplit 
words in the backup dictionary 245 (step 2305) . This 
procedure is similar to the procedure 1800 and can be 
understood with reference to Fig. 19, with the difference 
being that the computer now supplements the active 
vocabulary with the roots found for the unsplit words (at 
step 2300) . 

Split Unsplit Words (ste p 1715) 

Referring to Figs. 17 and 24, after the computer 
postulates the word fragments for unsplit words (step 1710) , 
the computer splits unsplit words in the backup dictionary 
according to a procedure 1715. Initially, the computer 
splits all of the words into component word fragments (step 
2405), as described below with reference to Fig. 25. The 
computer then counts the uses of each word fragment 1315 
(step 2410) . Finally, the computer retains the most useful 
word fragments (step 2415) . This determination of 
usefulness may be based strictly on the number of uses of 
each word fragment, the relative frequency of the words in 
which they are used, or some other appropriate criterion. 

Referring also to Fig. 25, the computer splits words 
by initially determining all possible splits for each word 
(step 2500) . Possible splits are considered for all word 
fragments; including word roots (w) , true roots (t) and 
affixes (p or s) . For example, possible splits may include 
such combinations as: ws, rs, pw, pr, or ppw. Some common 
combinations or patterns and the frequency with which they 
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occur in a vocabulary for U.S. English are shown in the 
graph 2600 c£ Fig. 26. Relative numbers and percentages o£ 
these patterns are graphed for each possible pattern. Some 
splits of words are highly unlikely, such as the pwssss 
pattern, which occurs only 0.001% of the time. Also, some 
patterns may be disallowed, such as a ps pattern, wh.ch rs 

not shown in Fig. 26. 

After determining possible splits, the computer 
loads all words from the active vocabulary 230 and all word 
fragments 1315 into a data structure, such as a Patricra 
Tre! (step 2506) . Then, to split each word in the backup 
dictionary 245, the computer sets an initial word .ndex k to 
.ero (step 2510) . The computer then begins an iterative 
loop in which the computer finds a word or word fragment 
that matches the beginning of the backup dictionary word 

nLxed by k (step 2515) . This step may be -P^^^ ^^^^ 
times to find an arrangement of words or word ^-^.en th t 
completely match the kth word. This arrangement xs referred 

to as a split. 

Each time that the computer adds a word or word 
fragment to the split, including when the computer adds the 
first word or word fragment to the split, the computer 
determines whether the split is valid (step 2520) . That is, 
the computer determines whether the one or more words or 
word fragments that make up the split are a valid 
arrangement of words or word fragments. For --P^^' - 
arrangement that starts with a suffix might be inva id, as 
might an arrangement that includes a prefix followed 

immediately by a suffix or a root followed by a P-^-^ " 
, the counter determines that the split is not valid, then 

the computer rejects the split (step 2525) . 

If the computer determines that the split is valid, 

then the computer determines if the split covers the entire 



word (step 2530) . If the split does not cover the entire 
word, the computer tries to add to the split a word or word 
fragment that matches the next portion of the kth word (step 
2515) . 

5 Upon determining that the split does cover the 

entire word (step 2530), or upon rejecting a split (step 
2525) , the computer determines if all possible splits have 
been examined for the kth word (step 2532) . If there are 
other possible splits, then the computer finds a further 
10 pronunciation match for the beginning of the kth word (2515) 
and repeats the process. 

If there are no other possible splits, then the 
computer increments the word index k to select the next word 
J in the backup dictionary (step 2535) . The computer then 

Id 15 checks to see whether k equals a maximum value such that all 
':D unsplit words have been split from the backup dictionary 245 

M (step 2540) . If there are still unsplit words that can be 

H split, then the computer finds partial pronunciation matches 

for the kth word (step 2515) . If there are no unsplit words 
20 that can be split, then the computer outputs a set of splits 
(step 2545) , The set of splits may be all of the splits 
generated or may be a best-scoring subset of the set of 
splits . 



Generate New Complete Dictation Vocabulary (step 1610) 
25 Referring again to Fig. 16A, after identifying the 

word fragments (step 1605) , the computer generates the new 
active vocabulary and the new backup dictionary (step 1610) . 
The new active vocabulary and the new backup dictionary are 
generated according to a procedure 1610 illustrated in Fig. 
30 27 . 

Initially, depending on the language that is 
modelled, the computer may limit a total number of word 
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fragments, Np, that may be added to the active vocabulary 
(step 2705) . Typically, is around a few thousand (4000) 
and therefore is much smaller than the size of the active 
dictionary, which is around 40,000. Next, the computer 
5 creates a temporary active vocabulary that includes the 

original active vocabulary, the Np word fragments, and a set 
of word roots (around a few thousand) that are in the backup 
dictionary (step 2710) . The computer then splits the words 
in this temporary active vocabulary and identifies as many 
10 of the splittable words as are necessary to reduce the size 
of the active vocabulary to a desired level (step 2715) . 
Thus, if the temporary active vocabulary has 47,000 
elements, then 7,000 splittable words may be identified to 
Q reduce the temporary active vocabulary to 40,000 elements. 

15 Once the computer identifies these splittable words, 

it trims the temporary active vocabulary by keeping word 
fragments and moving a set of splittable words into the 
backup dictionary 245 (step 2720). For example, if "cook", 
"cooks" , "cooker" , "cookie" , "book" , "books" , "booker" , 

0 2 0 "bookie", "+s", "+er", and "+ie" were in the temporary 
§ active vocabulary, then the splittable words "cooks", 
=5 "cooker", "cookie", "books", "booker", and "bookie" could be 

1 moved into the backup dictionary 245 and the words and word 
fragments "cook", "book", "+s", "+er", and "+ie" would be 

2 5 retained in the active vocabulary. The set of words and 
word fragments remaining after the size of the temporary 
active vocabulary is reduced to the desired level is 
designated as the new active vocabulary. 

After the active vocabulary is generated (step 
30 2720) , the computer splits words in the backup dictionary 
245 without using short roots (step 2725) . Then the 
computer splits unsplit words in the backup dictionary 245 
using short roots (step 2730). During these steps, for each 
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splittable word that has more than a single pronuno.at on, 
the computer may choose a single pronunciation to -P^^^ 
once each word has a single pronunciation and all splrttable 
words in the bac.up dictionary are split, the computer ma.e 
. list of word fragments 1315 that have not been used rn the 
splitting <step 2735) . This list may be used when the 
language model is created (step 1615) . 

^ v^,.-^ T,anau ° T- "'^'^''1 '«'''^P l"^' ' , ^. 

Referring to Fig. 28R, in one implementation, the 
computer creates the language model associated with the 
active vocabulary according to a procedure 1615A 
initially, the computer retrieves a training set of data, 
uch as I collection of text (step 2800) . The training 
collec ion of text includes words from the baCup dictionary 
and words from the active vocabulary. The computer modifies 
the collection of text by replacing any splittable backup 
dictionary words with their corresponding words and word 

. ,„t.n 2805) The computer then generates language 

fragments (step 2yub; . 

,11 elements of the active vocabulary -- 

model scores for all eiemeiiub ^ . 

: eluding words and word fragments - using ^ 
collection of text (step 2810) . For example, to produce a 
unigram language model, the computer counts the number of 
o Lrences of each word or word fragment in t- ""-^ 
collection of text and generates the language -^J^ 
based on these counts. Similarly, to produce a bigram 
language model, the computer counts the nu*er of 
occLrences of each consecutive pair of words or word 
Tragments in the training collection of text and generates 
the language model scores based on these counts. 

referring to Fig. 2SB, in another i-'P^--"";";^^ 
the computer creates the language model associated wit the 
active vocabulary according to a procedure 1615B. 
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procedure 1615A, the computer first retrieves the training 
collection of text (step 2825) . The computer then builds an 
N-gram language model, such as a unigram, bigram, or trigram 
model, using the retrieved training collection of text (step 
2 830) . The computer builds the N-gram language model based 
on how often each sequence of "N" words, referred to as an 
N-gram sequence, appears in the training collection of text. 

Next, for each N-gram sequence that includes one or 
more splittable backup words, the computer replaces any 
splittable backup words with their corresponding words and 
word fragments (step 2835) . This creates new N-gram 
sequences. For example, when the bigram "urgently go" is 
split into the sequence "urgent +ly go", two new bigrams 
("urgent +ly" and "+ly go") are created. As discussed in 
more detail below, the computer generates language model 
scores for the new N-grams based on the language model 
scores for the N-grams from which they were generated (step 
2840) . When splitting an N-gram results in an N-gram that 
exists already, the language model score for the existing N- 
gram is modified based on the language model score for the 
N-gram that was split. In general, the language model score 
for an N-gram is related to the probability that the N-gram 
will occur. 

Fig.- 2 8C shows an example of creating a unigram 
language model 2850. Two sample text segments that occur in 
the training collection of text are shown. A first segment 
2851 is "will urgently go where" and a second segment 2852 
is "to quickly go over". The count 2853 of the number of 
times that each unigram appears in the training text is also 
shown. In the sample text segments, two words may be split: 
"urgently" can be split into the word "urgent" and the 
suffix "+ly", and "quickly" can be split into the word 



"quick" and the suffix "+ly" to form split text segments 

2854 and 2855. 

Splitting of the sample text segments may result in 
a new count 2856 for each word or word fragment. For words, 
the count 2856 will reflect the previous count for the word 
plus the count of any additional instances of the word 
resulting from splitting another word. For example, the 
count 2856 for the word "will" equals the count 2853 for the 
word "will" (3550) plus where N„ni represents the total 

number of times that the word "will" occurs in a splittable 
word. For example, if the only splittable words in which 
the word "will" occurs are "willing", which occurs 500 
times, and "willful", which occurs 250 times, then N^i^ 
would equal 750. 

For word fragments, the count 2856 will reflect the 
combined counts of any splittable words that include the 
word fragment. For example, the count 2856 for the suffix 
"+ly" equals the count 2853 for "urgently" (500) plus the 
count 2 853 for "quickly" (350) plus N^y, where N^y represents 
the number of times that the suffix "+ly" occurs in other 
splittable words. For example, if the only other splittable 
word in which the suffix "+ly" occurs is "boldly", which 
occurs 200 times, then N^y would equal 2 00. 

Fig. 28D shows an example of creating a bigram 
language model 2875 using the training collection of text 
that includes segments 2851 ("will urgently go where") and 
2852 ("to quickly go over") . The count 2876 of the number 
of times that each bigram appears in the training text is 
also shown. As noted above, "urgently" can be split into 
the word "urgent" and the suffix "+ly", and "quickly" can be 
split into the word "quick" and the suffix "+ly" to form 
split text segments 2854 and 2855. 



Splitting of the sample text segments may result in 
a new count 2 877 for existing bigrams, as well as new counts 
for new bigrams. For existing bigrams, the count 2877 will 
reflect the previous count for the original bigram plus the 
count of any additional instances of the bigram resulting 
from splitting a word. For example, the count 2877 for the 
word "go" followed by the word "where" equals the count 2876 
for the word "go" followed by the word "where" (600) plus 
Ngowhere/ where Ngowhere is computed from the counts of bigrams 
that produce the word "go" followed by the word "where" as a 
result of the splitting. For example, if the splittable 
bigram "forego where", which occurs 2 5 times, splits into 
"fore+ go where", and is the only splittable bigram that 
produces the word "go" followed by the word "where", then 
Ngowhere would equal 25. 

For new bigrams, the count 2877 will reflect the 
combined counts of any splittable bigrams that include the 
new bigram. For example, the count 2877 for the suffix 
"+ly" followed by the word "go" equals the count 2876 for 
"urgently go" (200) plus the count 2876 for "quickly go" 
(100) plus Niygo, where N^yg^ is computed from the counts of 
bigrams that produce the suffix "+ly" followed by the word 
"go" as a result of the splitting. 

The method for computing the contribution of counts 
arising from the splitting of original bigrams to the counts 
for modified bigrams is as follows. The count for an 
original bigram is added to the count for a modified bigram 
if the modified bigram appears in the sequence of words and 
word fragments generated in the splitting of the original 
bigram, and the modified bigram contains all or part of the 
first word in the original bigram. 

For example, if the original bigram "urgently go" 
has a count of 200, and it splits into the sequence "urgent 



+ly go", then this original bigram contributes 200 to the 
counts of the modified bigrams "urgent +ly" and "+ly go", 
because both of these modified bigrams appear in the 
sequence generated by splitting "urgently go", and both 
contain a piece of the original first word ("urgently"). 

As another example, if the original bigram "to 
quickly" has a count of 150, and it splits into the sequence 
"to quick +ly" , then this original bigram contributes 150 to 
the count of the modified bigram "to quick", because "to 
quick" appears in the sequence generated by splitting "to 
quickly" and contains the original first word ("to"). 
However, no counts are contributed to the modified bigram 
"quick +ly", because even though it appears in the sequence 
generated by splitting "to quickly", it does not contain any 
piece of the original first word. This method of computing 
the contribution of original bigram counts to the counts of 
modified bigrams avoids excessive counting of modified 
bigrams . 

The devised method of counting bigrams may be 
generalized to the counting of modified n-grams from 
original n-grams as follows. The count for an original n- 
gram is added to the count for a modified n-gram if the 
modified n-gram appears in the sequence of words and word 
fragments generated in the splitting of the original n-gram, 
and the modified n-gram contains all or part of the first 
word in the original n-gram. 

Shown in Fig. 28E is a complete, but rather tiny, 
sample training collection of text that consists of 87 
words. Generally, however, the training collection of text 
contains orders of magnitude more words than the sample 
training collection of text shown in Fig. 2 BE contains. 

In this training collection of text, the unigram 
word sequence "whenever" occurs four times, the unigram word 
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active vocabulary. 
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D^r-fn^m Post-R e ^^nniMon Prnressinq (step 1660) 

once the speech recognizer 215 outputs the list of 
recognition candidates, the speech recognition system 100 
performs the post-recognition procedure 1660 illustrated xn 
Fig 29 in general, the system 100 . substitutes words from 
the backup dictionary or accepted by some set of rules for 
word fragments in a recognition candidate and rescores the 
recognition candidate xn view of the substitution. Rescored 
candidates are placed on a revised candidate list, 
candidates that do not include word fragments are 
transferred directly to the revised candidate list. Thus, 
the revised list may include both recognition candidates 
from the original list and rescored recognition candidates. 
TO begin the process, the system 100 retrieves a 
3 ^3 recognition candidate from the original list of candidates 
^ (step 2900) . If there are no word fragments in the 

recognition candidate (step 2905) , the system 100 transfers 
the original recognition candidate to the revised list of 

candidates (step 2910) . 

If there are word fragments in the recognition 
candidate (step 2905) , then the system 100 processes the 
word fragments (step 2915) . If any new recognition 
candidates were formed during the word fragment processing 
(step 2920) , the system rescores those recognition 
candidates (step 2925) . The system then adds any rescored 
candidates to the revised list (step 2930) . After the 
system adds a rescored or original candidate to the revised 
list or determines that no new candidates were formed m 
the word fragment processing, the system checks for more 
recognition candidates in the original list (step 2935) . I: 
there are more recognition candidates in the original list, 
then the system selects the next recognition candidate from 
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« 1-hat candidate (steps 
*e Ust (Step 2900, and processes that 



2905-2930) 



10 



15 



Prcces^iK^aaaa^^t^;^"^^ „ord fragments are 

Referring also to Fxg. 30 initially, the 

j'v.^ the procedure 2^1^- ^ 
processed ---^^^^Z;^ ^t/^^nce of word fragments or word 
system retrieves a first q candidate (step 

fragments and words from the „,,cord *ing 

3000, . using an earlier example, s PP ^ ^^^^^^^^ ^^^^^ 

is a recognition -"^^^^ ^ , likewise, other 

-^/» /"rlX word fragments for this 
sequences of words and/or ^^„g 
recognition candidate may be .mg 

-ly" ■ 100 determines whether the sequence 

Next, the system 100 de ^^^^^^ ^^^^^^ 

is an invalid sequence ^-^^I '^ J ^^^ ,i,,antly Incorrect 
an invalid sequence "^^^f^/^j.^/.^^rx. root/prefix, 
,,,acent co*lnations, such as 

suffix/root. -"^''/^"'i^^^L^fix may not he valid for the 
combinations, such as su ^^^^ separated and 

particular sequence, but may be ^^^^^ ^^^^^ ^^^^ 

„sed m other are never valid whether 

combinations, such as pr „ther sequences, 

taken alone or is encountered, the 

«hen such an inherently --^/^^^^^.^g^i.^on candidate and 
5 system may abort P— /"^ ^.^^^ conditions that may 
proceed to the next — ^^^^^^ ,,,rted are a 

cause processing of a rec g ^^^^ ^^^^ ^ p^^j^^. 

candidate that begins a sequence is 

".rthen retrieves the spelling rules 
30 valid, the system 100 the ^^^^^^^ ^^^^^ ^„ 

associated with each ^"''^ ^^^^^ cne spelling 

initial set of spelling rules that 
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rule for each affix in the sequence (step 3010) . For 
example, if the sequence is "accord +ing" , then each 
possible spelling rule set includes a single spelling rule 
corresponding to the suffix "+ing". Likewise, if the 
5 sequence is "accord -fing +ly" , then each possible spelling 
rule set includes a spelling rule corresponding to the 
suffix "+ing" and a spelling rule corresponding to the 
suffix "+ly" . 

The system 100 uses the spelling rule set to form a 
10 prospective word (step 3015) . For example, if the sequence 
is "accord -fing", and the spelling rule "-null, +ling" is 
applied, then the system 100 forms the prospective word 
"accordling" . Likewise, if the spelling rule "-null, +ing" 
.C3 is applied, then the system 100 forms the prospective word 

y 15 "according". 

\0 If the prospective word is in the active vocabulary 

(step 3 020) , then the recognizer should have found the word 
4 upon initial recognition, which indicates that the 

combination of word fragments and/or words to form the word 
0 2 0 may be an error. Accordingly, the system may not further 
process the prospective word. 

If the prospective word is not found in the active 
v3 vocabulary (step 3020), but is found in the backup 

dictionary or accepted by a set of rules (step 3025) , then 
25 the system generates a candidate using the prospective word 
(step 3030) . If the generated candidate includes additional 
word fragments (step 3032), the system processes the 
generated candidate using a recursive procedure comparable 
to the procedure 2915 (step 3034). 
3 0 After generating the candidate using the prospective 

word, after determining that the prospective word is not in 
the backup dictionary, or after determining that the 
prospective word is in the active vocabulary, and there are 
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more spelling rules associated with the affixes in the 
sequence (step 3035) , then the system gets the next set of 
spelling rules (step 3040) and, using the next spelling rule 
set forms a prospective word (step 3015) . 
5 ' If there are no more spelling rules associated with 

the affixes in the retrieved sequence (step 3035) , or if the 
retrieved sequence is invalid (step 3005) , and there are 
other possible sequences in the recognition candidate (step 
3045) , then the system retrieves the next sequence (step 
10 3000) and proceeds as discussed above. 

Other embodiments are within the scope of the 
following claims. For example, the techniques described 
here are not limited to any particular hardware or software 
configuration; they may find applicability in any computing 
15 or processing environment that may be used for speech 

recognition. The techniques may be implemented in hardware 
or software, or a combination of the two. Preferably, the 
techniques are implemented in computer programs executing on 
programmable computers that each include a processor, a 
20 storage medium readable by the processor (including volatile 
and non-volatile memory and/or storage elements) , at least 
one input device, and at least one output device. Program 
code is applied to data entered using the input device to 
perform the functions described and to generate output 
25 information. The output information is applied to one or 

more output devices. 

Each program is preferably implemented in a high 
level procedural or object oriented programming language to 
communicate with a computer system. However, the programs 
30 can be implemented in assembly or machine language, if 
desired. In any case, the language may be a compiled or 
interpreted language. 
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Each such computer program is preferably stored on a 
storage medium or device (for example, CD-ROM, hard disk or 
magnetic diskette) that is readable by a general or special 
purpose programmable computer for configuring and operating 
the computer when the storage medium or device is read by 
the computer to perform the procedures described in this 
document. The system may also be considered to be 
implemented as a computer-readable storage medium, 
configured with a computer program, where the storage medium 
so configured causes a computer to operate in a specific and 

predefined manner. 

Generation and use of word fragments (procedure 
1600) may include updating acoustic models in addition to 
updating language models. 

Hidden Markov models (HMMs) may be used to 
automatically split the pronunciation of all active words 
(or alternatively, all those words in the backup dictionary) 
into fragments or pieces, using, for example, a Baum-Welch 
training algorithm. This method may permit a complete 
bypass of the spelling rules if words are looked up in the 
dictation vocabulary using pronunciation rather than 
spelling. Because of this, pronunciations must be aligned 
with the spellings to determine the spelling rules. 

When the lookups are based on spelling rules, then 
recognition speed may be slowed down for long strings of 
roots and affixes. For example, each affix has a number of 
spelling rules that are combined with spelling rules for 
other affixes to form a large number of spelling rule sets, 
speech recognition speed may be increased by reducing the 
number of lookups during recognition and post-recognition 
processing. 

one way of doing this is to perform lookups using 
pronunciation of word fragments rather than spelling rules. 
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Since each affix has only one pronunciation, each set of 
word fragments will result in only one lookup per root 

pronunciation. 

A second way would be for the computer running the 
speech recognition to keep a record of recent lookups (for 
example, in a hash table) . Because most lookups are 
unsuccessful, it might be useful to keep only the 
unsuccessful lookups, rather than the successful lookups. 

A third way may employ an existential dictionary 
that includes a hash table having entries that are either 
zero or one depending on whether a word has been used. One 
may hash the backup words once or several times into this 
structure If a generated word is hashed into word 
fragments, and if any of the word fragments are not found in 
the existential dictionary, it is certain that the word is 
not in the backup dictionary. Existential dictionaries are 
usually used for applications such as spell -checkers m 
which there is a very small chance that a lookup will 
produce a false positive because it is important that the 
spell-checker not overlook a word which is spelled 
incorrectly. When the existential dictionary is used m the 
speech recognition application to reduce lookups, it does 
not matter if there are a substantial number of words which 
it incorrectly says may be in the backup dictionary because 
it is only important to reduce the number of lookups. 

After the computer identifies the splittable words 
(step 2715) , the computer may move extra words into the 
backup dictionary (step 2720) . The size of the active 
dictionary may be reduced at step 2720. Alternatively, 
splittable words may be retained in the temporary active 
vocabulary, that is, the temporary active vocabulary may not 
be trimmed. 



In another implementation, during generation of the 
complete dictation vocabulary (step 1610) , the computer may 
split words in more than one way. For example, the word 
"unerring" can be split either "un+ err +ing" or "unerr 
5 +ing", with each split representing a different 
pronunciation of the word. 

In a further implementation, recognition candidates 
output by either the speech recognition or post-recognition 
processing may be represented as a "net" rather than a list, 
10 to avoid redundant processing. 

In another implementation, during the splitting 
procedure 1715, the computer may apply different spelling 
rules to determine which types of splits are valid. For 
E3 example, a valid split may include up to eight words or word 

?2 15 fragments, including exactly one word root or true root. As 
v3 another example, recognition speed may be increased by 

~uj permitting a maximum of three words or word fragments in a 

S split. Furthermore, depending on the language, certain 

combinations may be allowed or disallowed. For example, in 

s 

O 2 0 the Germanic languages, the combination of a true root 
'^0 adiacent to a word root might be a valid combination. 

During post-recognition processing (step 1660) , the 
system 100 may skip the step of checking for the prospective 
word in the active vocabulary (step 3020) . The system 100 
25 may also (or alternatively) skip the step of checking for 
the prospective word in the backup dictionary (step 3025) . 
Likewise, the prospective word may be accepted or rejected 
based on other rules or based on no rules. 

Although a backup dictionary may be used to look up 
30 prospective words in the preferred embodiment, it is also 

possible to use a set of rules to determine what patterns of 
root and endings constitute a valid word. The set of rules 
may use part-of -speech constraints (in addition to other 
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constraints described with respect to the P«f--^ 
en^odi^ent) . For example, the prefix re. attaches to a 
verb the prefix un. attaches to an adjective, and the 
Treflx super, attaches to a noun. Likewise, the suffix .er 
fran for.s a verb into a noun, the suffix .tion -ansfo^s a 
verb into a noun, the suffix .able transforms a verb into an 
ad ective, and the suffix .i.e transforms an ^^^^ ^ 
a noun. Thus, "unrealizable" - un. real .ize .able (plus 
=„»l\inc, rule) is a valid prospective word, while 
!u„re:"ir„al' , "reconf usion" , and ..uperrealizable" would 

" '"^"rTspective words that are checked in the backup 
dictionary may be constrained based on a P~"";" 
the prospective word. For example, if the recognized words 
nl worl fragments include .rob. followed by .Ing, then two 
prospective words, namely "robing, and '^^^^ff^^^^ 
Lsslble If pronunciation is checked, in addition to 
ZZZ: then^ne of the words would be eliminated from 

,0 ™"kewise, prospective words Checked in the backup 

3 dictionary may be constrained based on --'^^^ ^ 

5 phonemes associated with the prospective word with words 

•4 the backup dictionary. 

^ in another implementation, the constituent 

fr-aaments 1315 may include syllables. A 
25 components or word fragments i^xd . ^ • „^ of a sinqle 

syllable is a unit of spoken language cons.stxng of a sxngle 

^interrupted sound formed by a vowel or diphthong alone, of 
unmterrupte ^.^^ ^^^^ 

a syllabic consonant alone, or ul 

consonants. Generally, a syllable includes a vowel and 
30 or more consonants clustered around the vowel. 

TO generate the syllables, it is necessary to relax 
the restriction that word fragments are actually in the 
Sctatfo: vocabulary. The syllables may be generated either 

- 75 - 



using rules for concatenation or by hand coding. 
Furthermore, syllables must be marked to distinguish them 
from words. The active vocabulary may include both words, 
syllable-words (that is, syllables that are words, such as 
"foot"), and true-syllables (that is, syllables that are not 
words, such as "tion". 

During speech recognition, if a syllable is 
detected, then that syllable is combined with an adjacent 
syllable or word to form the prospective word. Then, the 
prospective word may be looked up in the backup dictionary, 
in one implementation, there may be about 10,000 syllables 
in the active vocabulary along with the words in the active 
vocabulary. 

For example, the word "wrapper" may be represented 
as a combination of the syllable .wrap, and the syllable 
.er. or as a combination of the syllable .wra. and the 
syllable.. per.. In this case, it is preferable to use the 
former syllable combination because the syllable .er- is 
included in many words. As another example, the word 
"relativistic" may be represented as a combination of the 
following syllables in sequence: .rel- .a. -tiv .is. 
.tic. Because syllables are defined to be an uninterrupted 
sound, they may be freely concatenated, that is, without 
using spelling rules. This is in contrast to word fragments 
such as affixes, that require spelling rules to be 
concatenated. For example, splitting the above example of 
"wrapper" into roots and affixes requires the spelling rules 
that combine the word root "wrap" with the suffix ".er", 
such as -p and +pper or -nil and +per. 
What is claimed is: 



